1. Field of the Invention
The present invention relates to an image processing technique for reducing a chromatic aberration component and a blur component included in an image produced by image pickup.
2. Description of the Related Art
Images produced by image pickup apparatuses such as digital cameras include a blur component (image blur component) that is an image deterioration component caused by various aberrations of an image taking optical system (hereinafter simply referred to as an “optical system”) such as spherical aberration, comatic aberration, field curvature and astigmatism. Such a blur component is generated because a light flux emitted from one point of an object forms an image with some divergence on an image pickup surface, the light flux being normally converged at one point if there is no aberration or diffraction.
The blur component herein is optically expressed as a point spread function (PSF), and is different from blur caused by defocusing. Moreover, color blur in a color image caused due to longitudinal chromatic aberration, chromatic spherical aberration or chromatic comatic aberration of the optical system can be said to be a difference between blurring degrees of respective light wavelengths.
As a method for correcting the blur component of the image, there is known a correction method that uses information on an optical transfer function (OTF) of the optical system. This method is referred to as “image restoration”. Hereinafter, processing for correcting (reducing) the blur component of the image by using the information on the optical transfer function (OTF) of the optical system is referred to as “image restoration processing”.
The outline of the image restoration processing is as follows. When g(x, y) represents a degraded image (input image) including the blur component, f(x, y) represents an non-degraded original image, h(x, y) represents the point spread function (PSF) that forms a Fourier pair with the optical transfer function, * represents convolution, and (x, y) represents coordinates on the image, the following expression is established:g(x,y)=h(x,y)*f(x,y)  (1)
Moreover, converting the above expression into a form of a two-dimensional frequency surface through Fourier transformation shows the following expression of a form of a product for each frequency:G(u,v)=H(u,v)·F(u,v)  (2)where H indicates a result of Fourier transformation of the point spread function (PSF), in other words, an optical transfer function (OTF), and (u, v) indicates coordinates on the two-dimensional frequency surface, in other words, a frequency.
In order to acquire the original image from the degraded image, both sides of the expression only need to be divided by H as below:G(u,v)/H(u,v)=F(u,v)  (3)
Returning the F(u, v) through inverse Fourier transformation to a real surface enables acquisition of a restored image equivalent to the original image f(x, y).
When R represents a result of inverse Fourier transformation of H−1, performing convolution processing for an image in the real surface as represented by the following expression similarly enables acquisition of the original image.g(x,y)*R(x,y)=f(x,y)  (4).
This R(x, y) in the above expression is referred to as an “image restoration filter”. A real image includes a noise component, and hence use of the image restoration filter produced from a complete inverse of the optical transfer function (OTF) as described above results in amplification of the noise component together with the degraded image. Therefore, generally, a good image cannot be acquired. In this regard, there is known a method such as use of a Wiener filter for suppressing a high frequency side restoration rate of an image according to an intensity ratio of an image signal to a noise signal. Degradation of a color blur component of an image is substantially corrected by, for example, causing blur amounts of respective color components to be uniform by the correction of the blur component.
Japanese Patent Laid-Open No. 2007-028040 discloses a method of correcting color shift due to longitudinal chromatic aberration. Since peak positions of MTFs (modulation transfer functions) of the respective color components are different from each other due to the longitudinal chromatic aberration, the MTFs of the respective colors have differences. An image of a color component whose MTF is low is blurred with respect to an image of a color component whose MTF is high, which generates color blur. The method disclosed in the image of the Japanese Patent Laid-Open No. 2007-028040 estimates the image of the color component whose MTF is low, by using the image of the color component whose MTF is high, so as to eliminate the difference between MTF characteristics of the respective colors, thereby reducing the color shift.
Moreover, Japanese Patent Laid-Open No. 2008-085773 discloses the following method of correcting the color blur. This method moves an image pickup element and an optical system relatively to each other in an optical axis direction to perform image pickup plural times when image formation positions of lights of peak wavelengths in respective color components are located on the image pickup element so as to obtain plural color images. Then, the method combines the plural color images obtained by the image pickup into one image and then outputs it. This method makes it possible to eliminate differences between image formation characteristics of the color components to reduce the color blur in the combined image.
A correction filter used to correct the color blur is normally optimized for a state where an optical system of an image pickup apparatus is in an in-focus state (focused state). Therefore, applying the correction filter to an out-of-focus area (defocused area) in an image provides different correction from that for an in-focus area (focused area) in the image. For example, chromatic aberration in the in-focus area where an object distance is equal to an in-focus distance is different from that in the out-of-focus area where the object distance is different from the in-focus distance. In particular, when longitudinal chromatic aberration is generated, the respective color components are separated from each other to form MTF peaks in an MTF characteristic in the optical axis direction.
FIG. 9A shows separation of respective wavelength components (color light components) of white light emerging from an optical system 201 having longitudinal chromatic aberration. The respective color light components that have passed through the optical system 201 form images at mutually different positions, which results in formation of MTF peaks of respective colors each centering on an image formation position of each color light components as shown in FIG. 9B. An image acquired by an image pickup element 202 includes differences of the MTFs of the respective colors.
FIG. 10B shows an image restoration effect of an image restoration filter when being applied to an in-focus area of an image acquired by image pickup through an optical system having an MTF characteristic shown in FIG. 10A. When applying such an image restoration filter to an out-of-focus area of the image, MTFs of respective colors have differences as shown in FIG. 10D. This is because H−1 in the above-described expression (3) increases the MTF in the out-of-focus area by a same rate as that in the in-focus area. In the in-focus area and the out-of-focus area, shapes of OTFs and relationships of the OTFs of R, G and B components are mutually different, and therefore applying the image restoration filter to the out-of-focus area without change generates mismatch between the color components.
In comparison of FIG. 10C with FIG. 10A, the MTFs of the G and B components are deteriorated, but the MTF of the R component is improved. Thus, applying the same image restoration filter as that for the in-focus area to the out-of-focus area excessively increases an image restoration effect for the R component and decreases image restoration effects for the B and G components, which increases the differences between the MTFs of the respective color components as shown in FIG. 10D.
In this case, at an edge portion of an image shown in FIG. 11A, luminances of the G and B components are significantly different from that of the R component as shown in FIG. 11B, which causes a color blur-like appearance. Specifically, although the MTF of the R component is increased, the MTFs of the B and G components are kept to be low. This makes the B and G components noticeable at the edge portion to emphasize the color blur more than before the image restoration.
The method disclosed in Japanese Patent Laid-Open No. 2007-028040 estimates the image whose MTF is low by gradating the image whose MTF is high, and therefore resolution of the entire image is decreased.
Moreover, the method disclosed in Japanese Patent Laid-Open No. 2008-085773 sequentially performs image pickup plural times to acquire the plural color images, and therefore time differences are generated between the plural color images. This may cause changes in position, size and the like of the object between the respective color images. In this case, combining the plural color images without change generates color shift in the combined image. Moreover, magnifications of the respective color images are different from each other, which generates color shift due to chromatic aberration of magnification in a peripheral area of the combined image.
Furthermore, each of the methods disclosed in Japanese Patent Laid-Open Nos. 2007-028040 and 2008-085773 is applied to the color blur in the in-focus area, and therefore does not consider the color blur in the out-of-focus area. Thus, the method cannot reduce the color blur in the out-of-focus area after the image restoration.